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CHAPTER  I:  THE  PROBLEM 


Introduction 

You  will  not  find  it  difficult  to  prove  that 
battles,  campaigns,  and  even  wars  have  been 
won  or  lost  primarily  because  of  logistics. 

General  Eisenhower, 1945  (Daniel, 1947) 

Logistics  support  has  always  been  a  key  element  of 
combat  effectiveness  as  well  as  an  important  element  in 
commercial  transportation  systems.  The  U.S.  Air  Force, 
like  any  private  company,  has  a  need  to  manage  resources 
efficiently  and  to  assess  the  potential  capabilities  of 
their  limited  resources.  The  Air  Force  is  interested  in 
developing  tools  and  methods  to  assess  the  impact  of 
logistics  on  combat  capability.  This  research  will 
primarily  be  concerned  with  the  impact  of  aircraft  spares 
support  on  the  mission  success  of  military  combat  aircraft 
but  the  basic  methodology  should  be  extendible  to  commercial 
aircraft  fleets.  Several  Air  Force  organizations  have  an 
active  interest  in  this  area  since  they  each  have 
responsibilities  in  analyzing  the  efficiency  and 
effectiveness  of  Air  Force  resource  allocation.  They 
include  HQ  USAF/LEYS,  HQ  AFMC  WSMIS,  and  HQ  ACC/LGSW. 


□  □ 


Background 


The  following  is  a  brief  description  of  the  logistics 
assessment  problem.  During  day-to-day  flight  operations  and 
training,  peace-time  operating  stocks  (POS) ,  co-located  with 
the  unit,  are  used  to  provide  spares  support.  When 
hostilities  arise  however,  units  can  be  deployed  outside  of 
established  Air  Force  supply  chains.  In  order  to  plan  for 
these  contingencies,  additional  wartime  support  for  Air 
Force  units  is  provided  through  the  War  Reserve  Material 
(WRM)  program.  This  program  is  designed  to  support  deployed 
operating  units  and  relies  on  prepositioning  of  materials 
based  on  preplanned  programs  and  schedules  (AFR  400-24, 

1990) .  In  the  event  of  hostilities  the  War  Reserve  Material 
stock  is  additional  equipment  held  in  reserve  which 
supplements  normal  peacetime  operating  stocks  until 
industrial  production  can  sustain  combat  requirements.  It 
includes  spares,  equipment,  war  consumables,  medical 
material,  weapons,  and  other  material  designated  as  WRM  by 
Air  Force  Regulation  400-24.  The  Mobility  Readiness  Spares 
Package  (MRSP)  and  the  In-Place  Readiness  Spares  Package 
(IRSP)  are  a  part  of  the  War  Reserve  Material  program  for 
units  with  aircraft,  vehicles,  communication  systems  and 
other  specified  systems.  A  MRSP  is  defined  as  an  air- 
transportable  kit  of  critical  spare  parts  that  provides 
sustained  operations  during  wartime  or  contingency 
operations  when  normal  supply  channels  are  interrupted  or 


V 


fall  short  of  demand.  An  IRSP  performs  the  same  function 
but  it  is  meant  for  a  unit  which  will  stay  in-place  for  the 
conflict.  They  are  meant  to  sustain  a  unit  for  some 
specified  period  of  time  (usually  30  days)  without  external 
resupply  (AFR  67-1,  1993) .  When  resupply  of  these  spares 
packages  becomes  available,  it  arrives  from  two  sources. 

The  first  source  is  from  repair  in  the  field  and  the  second 
source  arrives  via  resupply  channels.  Items  that  are 
removed  from  aircraft  on  the  flightline  are  sent  to  unit- 
level  maintenance  for  repair  and  then  returned  to  supply. 
Items  broken  beyond  the  unit's  repair  capability  are  sent  to 
a  higher  level  and  a  new  part  is  ordered  (Demmy  and  Hobbs, 
1983) 

One  area  of  interest  is  the  management  and  assessment 
of  the  recoverable  spares  in  the  Readiness  Spares  Packages. 
Recoverable  spares  are  those  parts  in  the  aircraft  which  can 
be  repaired  and  reused.  To  model  the  impact  of  these  spares 
on  combat  capability,  the  Air  Force  uses  a  model  called 
Dynamic  Multi-Echelon  Technique  for  Recoverable  Item  Control 
(Dyna-METRIC) .  Dyna-METRIC  is  an  analytical  model  which 
runs  on  a  mainframe  computer  at  the  Air  Force  Materiel 
Command  Headquarters.  Although  Dyna-METRIC  is  a  flexible 
tool,  it  has  a  number  of  limitations.  This  has  reduced  it’s 
effectiveness  in  realistically  predicting  sortie  generation 
capability  (number  of  times  an  aircraft  can  fly  each  day) 
and  the  number  of  aircraft  combat  ready  especially  for  units 
with  fewer  than  12  aircraft  (Miyares,  1993) .  The  model 


assumes,  for  example,  that  unlimited  maintenance  personnel 
and  test  equipment  are  available,  that  all  aircraft  parts 
are  mission-essential,  and  that  all  aircraft  parts  fail  at 
the  same  Air  Force-wide  flying  hour-based  failure  rate. 

Also,  although  the  model  considers  the  impact  of  component 
redundancy,  this  capability  has  not  been  effectively  used 
(King,  1993) .  Due  to  these  shortcomings,  the  Air  Force 
wants  a  more  flexible  method  which  can  address  the  areas 
where  Dyna-METRIC  is  limited. 

The  original  work  in  this  area  assumed  that  all  demand 
processes  were  Poisson  with  a  mean-to-variance  ratio  of  one. 
Although  subsequent  work  has  relaxed  the  distribution 
constraint  to  allow  the  use  of  a  negative  binomial  or 
binomial  distribution,  the  Air  Force  is  still  using  the 
Poisson  distribution  for  demand  calculations.  Since  the 
demand  distribution  has  little  effect  on  this  research 
problem,  this  research  will  continue  to  use  the  Poisson 
distribution: 

p(x)  =  (mT)xe_nt  /  x!  x  =  0,  1,  2,...  (1-1) 

where:  m-  average  annual  demand 
T-  average  time  period 
x-  number  of  demands 

The  central  theorem  in  this  inventory  model  is  Palm's 
theorem  (Palm, 1934).  This  theorem  is  also  called  the 
infinite  channel  queuing  assumption  (Sherbrooke,  1992) . 


Palm's  Theorem:  If  demand  for  an  item  is 
given  by  a  Poisson  process  with  mean  m  per 
unit  time,  and  if  the  repair  time  for  each 
failed  unit  is  independently  and  identically 
distributed  according  to  any  distribution 
with  mean  repair  time  T,  then  the  steady- 
state  probability  distribution  for  the  number 
of  units  in  repair  has  a  Poisson  distribution 
with  mean  mT.  (Sherbrooke,  1992) 


When  the  model  deals  with  a  small  number  of  aircraft,  this 
theorem  is  violated  because  the  demand  distribution  and  the 
repair  cycle  are  no  longer  independent.  I  will  give  an 
example  later  which  will  demonstrate  this  problem. 

The  following  mathematical  development  was  adapted  from 
Sherbrooke  (1992) .  The  number  and  location  of  all  s  spare 
parts  can  be  shown  in  the  following  equation: 

s  -  OH  +  DI  -  BO  (1-2) 


where  OH-  The  number  of  spares  on  the  shelf 

DI-  The  number  of  items  due-in  from  repair  or 
resupply 

BO-  The  number  of  backorders  for  an  item 
s  -  The  stock  level  or  inventory  position 
This  is  a  balanced  equation  because  the  order  quantity  is 
assumed  to  be  one.  Any  change  in  one  variable  will  result 
in  a  change  in  another.  For  example,  if  a  demand  occurs, 
the  number  due-in  will  increase  by  one  and,  depending  on  the 
current  on-hand  balance,  the  on-hand  balance  will  either 


decrease  by  one  or  the  number  of  backorders  will  increase  by 
one. 

One  goal  of  the  model  is  to  predict  the  number  of 
aircraft  available  and  the  number  of  sorties  that  can  be 
flown  during  a  specified  time  period  based  on  an  initial 
spares  position.  In  order  to  do  this  the  model  calculates 
the  number  of  expected  backorders  EBO(s)  for  each  item  on 
the  aircraft  based  on  an  initial  stock  position.  This  is 
done  using: 

EBO(s)  -  Pr[DI=s+l]+  2  Pr[DI=s+2]  +  3  Pr [DI=s+3] + . . . 

oo 

=  £(x  -  s)  Pr [DI  =  x]  (1-3) 

X=Bf  1 

Another  term  used  is  pipeline.  It  represents  the  number  of 
units  of  a  part  in  repair  at  a  location  or  in  the  resupply 
chain.  The  average  pipeline  is  the  average  demand  times  the 
repair  time  n  =  mT  for  the  single  base,  full  repair  ,  no 
depot  resupply  case.  As  a  result  of  Palm's  Theorem,  the 
average  pipeline  value  becomes  the  mean  of  the  Poisson 
distribution  for  calculating  the  expected  backorders.  If  we 
allow  multiple  bases,  limited  base  repair,  and  depot  repair 
and  resupply,  the  average  pipeline  at  base  j  becomes: 

Uj  =  m-jO-ft  +  (1  -  rj)  [Oj  +  EB0[so|moTo]  /  mQ] )  (1-4) 


where  mj  =  average  annual  demand  at  base  j 


Tj  =  average  repair  time  at  base  j 
Hj  =  average  pipeline  at  base  j 
Tj  =  probability  of  repair  at  base  j 
Oj  =  average  order  and  ship  time  from  depot  to 
base  j 

subscript  j  =  base  counter 
subscript  0  =  depot  counter 

For  most  aircraft  combat  assessments  this  equation  reduces 
to  m  =  (TjTj)mj  since  depot  repair  and  resupply  are  not 
available.  As  stated  earlier,  this  value  becomes  the  mean 
of  the  Poisson  distribution  used  to  calculate  the  expected 
backorders.  Since  aircraft  availability  (number  of  aircraft 
available  versus  the  number  of  aircraft  fielded  times  100) 
is  one  of  the  important  Air  Force  measures  of  merit,  it  can 
be  calculated  from  the  expected  backorders  as  follows: 

I 

A  =  100X  [1  '  EBO^Si)  /  (NZi)  (1-5) 

subject  to  -  EBOi(Si)^  NZi  for  every  i 

where  Zj  -  number  of  times  the  same  item  occurs  on  a 
single  aircraft 

N  -  number  of  aircraft  fielded 

This  formula  implies  that  an  aircraft  is  available  only  if 
there  are  no  holes  in  any  of  the  Z±  locations  on  the 


aircraft.  The  constraint  simply  prevents  the  number  of 
backorders  from  exceeding  the  number  of  possible  aircraft 
holes.  The  number  of  predicted  aircraft  flights/flying 
hours  is  simply  the  number  of  available  aircraft  times  the 
maximum  flight  rate  which  is  then  capped  by  the  number  of 
flights  required  (i.e.  Available  Aircraft  *  max.  flight  rate 
=  number  of  sorties  flown,  capped  at  the  daily  required 
amount)  . 

The  following  numeric  example  displays  the  mathematical 
problems  that  occur  when  the  model  attempts  to  assess  the 
combat  capability  of  a  small  number  of  aircraft.  Assume  our 
unit  has  only  one  aircraft  with  2  parts  A  and  B.  The 
following  information  is  known: 


\ 


part  a  part  b 

Demand  rate  (dmd/flyhr)  0.05  0.03 

%  of  base  repair  100  100 

repair  cycle  time (days)  2  3 

on-hand  stock  0  0 


Based  on  the  above  data  with  a  requirement  to  fly  one 
aircraft,  once  for  eight  hours  each  day  for  10  days,  the 
model  predicts  the  following: 

part  a  part  b 

EBOs  0.8  0.72 

Pipeline  0.8  0.72 

Availability  =  5.6% 

This  implies  that  you  can  expect  to  fly  0.056  sorties/day, 
have  0.056  available  aircraft  on  day  10,  and  fly  4.48  hours 
for  the  whole  10  day  period.  A  simulation  using  tha  same 
input  data  predicted  a  sortie  capability  of  0.47  sorties/day 
and  an  average  of  38.2  flying  hours  during  the  10  day 
period.  The  simulation  model  seems  to  give  a  much  better 
picture  of  what  might  really  happen  because  the  METRIC  model 
continues  to  break  parts  even  when  flying  hours  are  not 
being  accumulated. 


Research  Objective 


Develop  a  methodology  to  assess  the  impact  of  spares 
support  on  combat  aircraft  effectiveness  when  deployed  as  a 
small  unit  with  a  Mobility  Readiness  Spares  Package  (MRSP) . 

Sub-Ob j actives 

-  Clearly  define  what  a  small  unit  is  (ie.  where  do  the 
Dyna-Metric  assumptions  begin  to  break  down  based  on  number 
and  complexity  of  aircraft) - 

-  Build  a  user-friendly  data  interface  for  input  data 
manipulation. 

-  Document  the  causes  of  the  current  methods's 
inadequacies. 

-  Investigate  other  approaches  for  assessing  the  impact  of 
spares  support  on  aircraft  combat  effectiveness. 

-  Develop  a  methodology  that  addresses  the  analytic 
shortcomings  of  Dyna-METRIC  to  include  redundancy,  item 
importance,  and  Sortie/Fully  Mission  Capable  (FMC)  aircraft 
calculations . 


-  Validate  the  methodology  using  appropriate  aircraft 
types  and  sizes  and  compare  with  actual  experience. 

Scope  and  Limitations 

This  research  is  centered  around  trying  to  find  a  way 
to  more  accurately  predict  the  impact  of  sparing  policies  on 
the  combat  effectiveness  of  a  small  deployed  or  isolated 
unit.  The  following  limits  are  assumed: 

1)  Only  one  aircraft  type  and  unit  will  be  analyzed  at 
a  time. 

2)  All  of  the  aircraft  are  deployed  or  locate  dat  the 
same  location  so  that  transportation  and  ordering  times  of 
spares  from  supply  are  not  significant. 

3)  Resupply  from  outside  sources  is  not  allowed  during 
the  analysis  period. 

4)  The  parts  structure  has  only  two  levels  of 
indenture  known  as  line  replaceable  units  (LRU)  and  shop 
replaceable  units  (SRU) . 


Chapter  II:  Literature  Review 


METRIC  Models 

The  purpose  of  this  literature  review  is  to  look  at  the 
history  of  Multi-Echelon/  Multi-Indenture  Inventory  models 
with  a  particular  emphasis  on  the  development  of  the  Dynamic 
Multi-Echelon  Technique  for  Recoverable  Item  Control  (Dyna- 
METRIC)  model  that  is  currently  being  used  by  the  Air  Force 
to  assess  weapon  system  wartime  capabilities.  The  basic 
theorem  for  most  of  the  work  in  this  area  is  Palm'  s  Theorem 
on  queuing.  This  theorem  is  stated  below: 


Palm's  Theorem:  If  demand  for  an  item  is 
given  by  a  Poisson  process  with  mean  m  per 
unit  time,  and  if  the  repair  time  for  each 
failed  unit  is  independently  and  identically 
distributed  according  to  any  distribution 
with  mean  repair  time  T,  then  the  steady- 
state  probability  distribution  for  the  number 
of  units  in  repair  has  a  Poisson  distribution 
with  mean  ml.  {Sherbrooke, 1992) 


This  theorem  is  important  because  it  states  that  the  shape 
of  the  repair  time  distribution  with  specified  mean  T  has  no 
impact  on  the  steady-state  probability  distribution  of  the 
number  of  units  in  repair.  The  steady-state  distribution 
for  the  number  of  items  in  repair  is  going  to  be  Poisson 
with  mean  MT  (Sherbrooke, 1992) .  The  proof  of  Palm's  Theorem 
is  adapted  from  Hadley  and  Whitin  (1963)  and  presented 
below: 


PROOF: 


Assume  r(p)  is  the  probability  that  the  repair  time  is 
p  when  the  mean  repair  time  is  T.  The  probability  of  a  unit 
completing  repair  at  time  t,  having  started  at  ti  where  t  > 
ti  is  shown  below: 

t-ti 

R(t  -  J  r  ( p )  dp  (2_i) 

o 

If  at  least  one  demand  has  occurred  in  the  interval  0  to  t, 
then  the  probability  that  a  unit  is  repaired  by  t  is: 

[R  ( t  -  ti)  /  t  ]  dti  (2-2) 

The  probability  that  a  specific  demand  between  0  and  t  is 
repaired  by  t  is  given  by: 

l/tjR(t-t)dt -[l/t]|R(p)dp  (2-3) 

o  o 

Let  s  be  the  number  of  items  available  initially  and  n 
be  the  net  inventory  where  the  net  inventory  is  the  current 
stock  balance  minus  any  backorders.  We  want  to  show  that  as 
t  goes  to  infinity,  the  number  of  items  in  repair  has  a 
Poisson  distribution  with  mean  mT.  First,  we  need  the 
probability  that  the  net  inventory  at  time  t  is  i  len  there 


have  been  z  demands  since  time  0.  The  values  of  n  range 
from  s  to  s  -  z  where  n  *  s  means  all  z  items  have  been 
repaired  and  n  -  s  -  z  means  that  none  of  the  z  items  have 
been  repaired.  This  implies  that  s  -  n  items  still  need  to 
be  repaired  at  time  t. 

If  there  have  been  z  demands,  there  must  be  2  +  s 
finished  repairs.  The  following  binomial  distribution 
represents  the  probability  that  s  -  n  repairs  have  yet  to  be 
completed  from  the  z  demands: 

b*n(n)  =  [S_Z  J(1  / 1)  j  R(p)dp 


(l/t)f{l-R(p)}dp 


(2-4) 


By  weighting  (4)  by  the  probability  of  having  z  items 
demanded  and  summing  over  all  z  2:  s  -  n  we  get  the 
unconditional  probability  that  the  net  inventory  is  n  at 
time  t  or: 


Pr[net  inventory  is  n  at  time  t]  =  p(zjmT)bin(n) 

T=Wr-ti 


I 


(2-5) 


Since  we  are  interested  in  the  limit  of  Pr[net  inventory  is 
n  at  time  t]  as  t-»oo,  notice  that: 

Urn  J  {l  -  R(p)}dp  =  |{l  -  R(p))dp 

0  0 


-J-pd[l-R(p)] 

0 

=  Jp(dR/dp)dp 
0 
m 

=  Jpr(p)dp  =  T  (2-6) 

o 

Substituting  this  into  equation  (5)  gives: 

lim  Pr[net  inventory  is  n  at  time  t]  =  poissonjs  -  nJmT}  ( 2  -  7 ) 

Because  s  -  n  is  the  number  of  items  being  repaired  with  a 
range  of  0  to  oo,  Palm's  Theorem  is  proved. 

According  to  Demmy  and  Hobbs  (1983),  beginning  with  the 
work  of  Feeney  and  Sherbrooke  in  1966,  optimization 
techniques  for  stationary,  multi-echelon,  multi-indenture 
inventory  and  repair  systems  which  use  ( s-l,s )  inventory 
policies  have  really  progressed.  Feeney  and  Sherbrooke 
(1966)  describe  the  (s-l,s)  inventory  system  where  the  item 
demands  can  have  any  compound  Poisson  distribution. 

Inventory  performance  in  this  situation  is  dependent  solely 
on  the  spare  stock  level,  s,  which  provides  protection 
against  stockouts.  If  delay  times  such  as  resupply  or 
repair  were  zero,  then  spare  stock  would  be  unnecessary. 
This,  however,  is  seldom  true  so  optimal  sparing  levels  are 
usually  greater  than  zero.  They  define  the  (s-I,s)  policy 
as  a  continuous  review  policy  where  a  demand  for  D  units 
results  in  an  immediate  reorder  of  D  units.  This  keeps  the 


total  on-hand  stock  plus  stock  on-order  minus  backorders 
equal  to  the  spare  stock  level  s.  The  objective  is  to 
compute  the  steady-state  probability  distribution  for  the 
number  of  items  in  resupply.  First,  they  generalize  and 
extend  Palm'  s  theorem  to  include  demands  that  have  a 
compound  Poisson  distribution.  They  provide  a  proof  of  this 
extension  in  an  appendix  to  the  article.  In  their  work, 
they  study  two  cases:  the  backorder  case  and  the  lost  sales 
case.  For  this  dissertation,  only  the  backorder  case  is 
important.  They  provide  three  measures  of  merit: 

1)  Ready  Rate  -  the  probability  that  a  unit  has  no 
backorders  when  reviewed  at  random  points  in  time. 

2)  Fills  -  the  expected  number  of  demands  that  can  be 
filled  immediately  from  stock  during  a  fixed  period  of  time. 

3)  Units  in  service  -  the  expected  number  of  items  in 
the  resupply  pipeline  at  any  given  point  in  time. 

They  also  provide  algorithms  and  FORTRAN  code  to  calculate 
these  performance  measures  in  the  appendices  to  the  article. 

In  1966  Sherbrooke  (1966)  wrote  the  Multi-Echelon 
Technique  for  Recoverable  Item  Control  (METRIC)  model.  It 
originally  had  three  purposes:  1)  Optimization  of  base  and 
depot  stock  levels  subject  to  cost  or  performance 
constraints,  2)  Redistribution  of  current  stock  levels 
between  base  and  depot  locations  to  provide  optimal  system 
performance,  and  3)  Assessment  of  system  performance  based 
on  current  stock  levels  at  each  location  against  a  selected 


scenario.  Sherbrooke  also  presents  several  general  model 
assumptions  that  he  says  are  not  exactly  true  but  provide 
good  approximations: 

1)  System  Objective  is  to  minimize  the  sum  of 
backorders  across  all  items  at  all  bases  for  a  weapon 
system. 

2)  The  demand  for  each  item  is  a  logarithmic  Poisson 
process . 

3)  Demand  is  stationary  over  the  prediction  period. 

4)  Decisions  on  where  to  repair  an  item  depend  only  on 
the  difficulty  of  repair 

5)  Lateral  Resupply  is  not  allowed. 

6)  No  condemnations  are  considered. 

7)  The  Depot  doesn't  batch  items  for  repair. 

8)  Allows  different  priorities  by  base  and  item  but 
not  within  an  item. 

9)  Demand  data  across  different  locations  can  be 
combined. 

For  each  of  these  assumptions  he  provides  support  for  their 
acceptance.  He  goes  on  to  provide  some  mathematical 
development  and  theory  for  the  model  and  suggests  methods 
for  collecting  and  analyzing  input  data.  He  closes  by 
presenting  several  applications.  They  include  cost- 
effectiveness  decisions,  minimum  stock  levels,  maximum  stock 
levels,  base  reparable  percentage  estimation,  and  average 
base  repair  time  estimation.  Sherbrooke  (1968b)  also 


provided  a  non-technical  management  version  of  this  report 
in  1968.  He  (Sherbrooke, 1968a)  then  published  a  shorter 
technical  version  in  Operations  Research.  In  1971, 
Sherbrooke  (1971)  again  wrote  about  his  METRIC  model.  He 
discusses  the  METRIC  evaluation  criteria  of  minimizing 
expected  base  backorders  subject  to  a  budget  constraint  and 
its  superiority  to  fill  rate,  the  number  of  units  supplied 
divided  by  the  number  demanded.  But,  he  points  out,  these 
are  still  both  measures  of  supply  and  only  indirectly 
operationally  relevant.  He  recommends  using  the  number  of 
aircraft  Not  Operational  for  Supply  (NORS)  as  a  more 
relevant  measure  of  merit.  The  problem  with  using  NORS, 
however,  is  that  a  model  to  minimize  NORS  aircraft  is  not 
mathematically  tractable  because  the  objective  function  is 
not  a  separable  function  of  item  performance  measures. 

Also,  along  with  some  of  the  mathematics,  he  presents  a  case 
study  of  the  Fill  aircraft.  He  uses  this  case  study  to 
demonstrate  the  effectiveness  of  the  METRIC  model.  First, 
he  computes  an  Fill  package  of  spares  using  the  METRIC  model 
subject  to  two  budget  constraints  of  3.49  and  3.89  million 
dollars.  Next,  he  evaluates  these  packages  using  NORS 
aircraft  as  the  measure  of  merit.  Then  experimenting  with 
other  stocking  policies,  he  tried  to  obtain  spares  packages 
that  gave  better  NORS  values  with  the  same  budget 
constraints.  After  many  trials,  the  best  improvement  over 
the  METRIC  model  performance  in  terms  of  NORS  aircraft  was 
less  than  1  percent.  He  considers  this  reassuring  but  warns 


that  the  results  are  empirical  and  based  on  only  one  test 
sample.  Also,  additional  policy  trials  may  still  be  able  to 
reduce  the  NORS  value  although  he  does  not  think  much 
improvement  will  occur.  This  is  the  first  in  a  series  of 
models  that  will  eventually  form  the  basis  of  the  Air  Force 
aircraft  recoverable  spares  assessment  and  requirements 
computations . 

In  1973,  Muckstadt  wrote  the  MOD-METRIC  model.  This 
extended  the  original  work  of  Sherbrooke  by  allowing  multi- 
indentured  items  to  be  considered.  Muckstadt  (1976) 
continued  his  work  and  in  1976  wrote  the  Consolidated 
Support  Model  (CSM) .  It  was  an  extension  of  the  MOD-METRIC 
model  that  allowed  a  three  echelon  system  to  be  analyzed 
versus  MOD-METRIC's  two  echelons.  This  model  also  added  an 
additional  constraint  or  assumption.  Every  line  replaceable 
unit  (LRU)  fixed  at  the  intermediate  repair  facility  can 
have  no  more  than  one  broken  shop  replaceable  unit  (SRU) . 

The  model  allowed  users  to  look  at  requirements  for  not  only 
assemblies  or  line  replaceable  units  (LRU)  but  also  for  the 
subassemblies  or  shop  replaceable  units  (SRU) .  His  article 
goes  on  to  describe  the  mathematical  development  of  this 
extension  and  provides  an  algorithm  for  determining  stock 
levels.  He  then  uses  this  algorithm  to  compute  stock  levels 
for  jet  engines  at  base  and  depot  level.  Muckstadt  also 
points  out  that  although  he  has  included  only  two  levels  of 
indenture,  the  analysis  method  is  easily  extended  to 
additional  levels  of  indenture. 


Finally,  in  1980  Hillestad  and  Carrillo  (1980)  made  a 


significant  breakthrough  by  finding  a  method  to  model 
nonstationary  demand  and  service  rates.  This  was  important 
because  assuming  stationary  demand  patterns  when 
transitioning  from  peace  time  to  war  time  was  not  considered 
appropriate.  This  particular  work  is  very  mathematical  and 
theoretical.  In  it  they  were  able  to  produce  many  time 
dependent  measures  of  system  performance  and  derive 
transient  results  for  nonstationary  distribution  periods. 
These  results  laid  the  groundwork  for  the  Dynamic  Multi- 
Echelon  Technique  for  Recoverable  Item  Control  (Dyna-METRIC) 
model.  Hillestad  (1982)  then  wrote  the  Dyna-METRIC  manual 
which  provides  a  less  rigorous  mathematical  description  of 
the  model  along  with  a  description  of  its  capabilities  and 
basic  assumptions.  Deming  and  Hobbs  (1983)  provide  a  brief 
but  functional  comparison  of  Dyna-METRIC  and  MOD-METRIC 
showing  that  they  have  similar  theoretical  foundations  but 
Dyna-METRIC  has  several  features  that  MOD-METRIC  does  not. 

It  can  consider  three  supply  echelons  versus  only  two  in 
MOD-METRIC.  Dyna-METRIC  can  also  estimate  ready  rates, 
sortie  rates,  and  other  measures  of  aircraft  readiness.  The 
only  drawback  for  this  expanded  capability  is  Dyna-METRIC' s 
inability  to  optimize  system-wide  spares  levels  which  MOD- 
METRIC  can  do.  Dyna-METRIC  can  only  optimize  at  the  base 


level. 


According  to  Pyles  (1984),  Dyna-METRIC  was  developed  to 
provide  five  new  pieces  of  logistics  information  for 
decision  makers.  They  include: 

1)  Operational  performance  measures. 

2)  Effects  of  wartime  dynamics. 

3)  Effects  of  repair  capacity  and  priority  repair. 

4)  Problem  detection  and  diagnosis. 

5)  Assessments  or  requirements. 

He  also  gives  examples  of  the  typical  Air  Force  performance 
measures  for  this  time  period  which  include: 

1)  Resource  Counts  (ex.  shelf  stock  and  War  Reserve 
Material  (WRM) ) 

2)  Process  delay  times  (ex.  repair  time  and  order  and 
ship  time) 

3)  Peace  time  customer  satisfaction  proxies  (ex. 
percent  requisitions  filled.  Not  Mission  Capable  (NMC) 
aircraft,  and  cannibalization  rates) 

Then  he  points  out  that  although  each  of  these  measures 
provides  some  measure  of  overall  support  for  U.S.  Air  Force 
systems,  there  is  no  integrated  method  to  assess  overall 
support  or  even  the  relative  importance  of  individual 
components.  This,  he  claims,  is  where  Dyna-METRIC  can 
provide  support.  It  ties  the  resource  counts  and  process 
delay  times  to  the  important  operational  concerns  of  number 
of  sorties  flown  and  number  of  fully  mission  capable  (FMC) 
aircraft  available  and  can  do  all  of  this  under  dynamic 
wartime  scenarios.  The  report  also  provides  an  extended 


example  of  how  to  use  the  model  for  a  single  FI 6  squadron 
deploying  to  a  single  base.  Pyles  also  provides  a  list  of 
Model  limitations,  a  description  of  each  limitation  and  its 
impact  on  modeling.  The  limitations  are: 

1)  Repair  procedures  and  productivity  are 
unconstrained  and  stationary. 

2)  Forecast  sortie  rates  do  not  reflect  flight  -  line 
resources  and  the  daily  employment  plan. 

3)  Component  failure  rates  vary  only  with  the  user 
defined  flying  program. 

4)  Aircraft  at  each  base  are  basically  the  same. 

5)  Repair  decisions  and  actions  happen  after  component 
testing. 

6)  Item  failure  rates  are  not  adjusted  at  based  on 
previous  sorties  completed. 

7)  The  repair  processes  are  the  same  at  every 
echelon's  repair  facility. 

Gage  and  Ogan  (1983)  give  some  additional  limitations 
that  are  important.  For  example,  the  actual  sorties  flown 
by  the  model  can  never  exceed  the  requested  sorties.  Also, 
the  Not-Mission  Capable  due  to  Supply  (NMCS)  figure  given  by 
the  model  is  not  necessarily  equivalent  to  the  number  of 
grounded  aircraft.  It  simply  means  the  aircraft  is  not  able 
to  accomplish  all  of  it's  required  missions.  Put  simply, 
the  model  does  not  consider  Partially  Mission  Capable  (PMC) 
aircraft.  The  model  also  still  carries  the  assumptions  that 
the  repair  and  demand  processes  are  independent,  the  depot 


is  an  unconstrained  source  of  supply,  and  the  intermediate 
repair  facility  distributes  parts  based  on  base  cumulative 
flying  hours.  Gage  and  Ogan  see  Dyna-METRIC  as  a  valuable 
management  tool  but  warn  that  it's  not  the  final  truth. 
Because  of  the  model  assumptions  and  limitations,  they 
recommend  continual  review  of  the  outputs  for  validity  and 
reasonableness . 

The  next  step  in  the  evolution  of  the  multi-echelon, 
multi- indenture  modeling  effort  came  in  the  form  of  VARI- 
METRIC  (Sherbrooke,  1986) .  Since  that  time,  several 
investigators  have  developed  exact  solutions  for  this 
problem  but  they  require  more  restrictive  assumptions  and 
substantially  more  computer  time.  Slay  (1980)  developed 
VARI-METRIC  and  Graves  improved  upon  it.  Graves  (1985) 
showed  that  in  11%  of  the  cases,  METRIC  computed  stock 
levels  varied  by  at  least  one  unit  from  the  optimal  while 
VARI-METRIC  only  differed  in  1%  of  the  cases.  This 
improvement  is  achieved  by  taking  into  account  not  only  the 
mean  pipeline  value  as  the  previous  models  did  but  also 
looking  at  the  variance  of  the  pipeline  quantities. 
Sherbrooke  (1986)  shows  that  in  the  case  of  multi- indenture, 
multi-echelon  systems,  VARI-METRIC  gives  estimates  of 
backorder  quantities  that  are  very  close  to  those  calculated 
by  simulation  models.  Thus,  he  believes  that  VARI-METRIC  is 
an  improvement  over  MOD-METRIC  and  Logistic  Managements 
Institute's  Aircraft  Availability  Model.  To  back  up  this 


claim,  he  presents  several  case  study  comparisons  of  the 
VARI -METRIC  and  MOD-METRIC  models. 

Dyna-METRIC  version  3.04  became  the  standard  aircraft 
unit  assessment  system  in  1988.  It  fell  under  the  umbrella 
of  the  Weapon  System  Management  Information  System  (WSMIS) 
of  the  Air  Force  Logistics  Command  (AFLC)  now  known  as  the 
Air  Force  Material  Command  {Isaacson  et.  al.,  1988). 

This  model  represented  a  full  scale  implementation  of 
the  model  discussed  in  Hillstad  and  Carrillo  (1980), 

Hillstad  (1982),  and  Pyles  (1984).  Version  4  is  the  current 
version  being  used  in  the  Air  Force  for  weapon  system 
assessment  (Isaacson  et.  al.,  1988).  It  was  developed  to 
assess  worldwide  logistics  support  of  aircraft  spares 
including  the  depot-theater  interactions.  This  allows 
someone  to  assess  movement  of  spares  from  one  theater  to 
another  or  to  assess  the  impact  of  base/theater/depot  repair 
processes,  stock  levels,  transportations  processes, 
cannibalization  policies,  and  wartime  plans  interactions  on 
the  military's  combat  capability.  It  also  allows  the  user 
to  view  how  spares  support  for  subcomponents  may  impact 
combat  capability  by  showing  the  impact  of  these  parts  in 
the  repair  process  at  each  echelon.  Isaacson,  Boren,  Tsai, 
and  Pyles  (1988)  give  a  detailed  description  of  the 
differences  between  version  3.04  and  version  4.  As 
mentioned  earlier,  one  assumption  of  the  METRIC-type  models 
is  unconstrained  repair  facilities.  In  1989  Tsai  (1989) 
wrote  Dynamic  Simulation  of  Constrained  Repair  (Dyna-SCORE) 


to  try  to  study  the  impact  of  maintenance  policies  on  weapon 
system  availabilities.  The  model's  outputs  include 
summaries  of  job  processing  times,  component  pipeline 
contents,  backorder  quantities,  weapon  system 
availabilities,  and  equipment  utilizations.  The  problem 
with  Dyna-SCORE  is  that  it  is  a  single-echelon  model  that 
focuses  on  only  one  repair  shop  and  assumes  that  all  other 
shops  and  echelons  have  no  impact  on  aircraft  availability 
except  to  fill  requisitions  or  create  demands  for  the 
modeled  shop  (Tsai,  1989) .  Dyna-METRIC  version  5  (Isaacson 
and  Boren,  1988)  was  an  attempt  to  extend  the  model  to 
consider  constraining  or  controlling  repair  processes  and 
uncertainties.  It  attempted  to  do  this  by  replacing  the 
pipeline  component  calculations  based  on  Palm's  Theorem  with 
a  Monte  Carlo  simulation.  Its  biggest  limitation  was  that 
in  an  attempt  to  model  constrained  repair  it  does  not  model 
part's  subcomponents.  This  implies  that  a  part  will  never 
be  delayed  in  a  repair  facility  while  it  waits  for  parts 
(Isaacson  and  Boren,  1988) .  Version  6  released  in  1993  is 
an  enhancement  of  version  5.  It  was  an  attempt  to  place 
back  in  the  model  the  version  4  features  that  had  been  left 
out  of  version  5.  It  still  uses  Monte  Carlo  simulation  for 
determining  pipeline  quantities  and  attempts  to  fix  the 
version  4  problem  of  computing  pipeline  demands  based  on 
scheduled  flying  hours  rather  than  actual  flying  hours.  Its 
most  significant  limitations  include  an  inability  to  compute 
spares  requirements  and  very  long  run  times  (Isaacson  and 


Boren,  1993) .  Two  other  versions  of  the  model  should  also 
be  mentioned.  Dyna-METRIC  Microcomputer  Analysis  System 
(DMAS)  is  a  microcomputer  version  of  the  Dyna-METRIC  version 
4  model  used  by  the  Air  Force.  It  has  restricted 
capabilities  and  is  primarily  intended  for  base-level  users 
who  are  doing  unit  level  requirements  and  assessment 
calculations  (DRC,  1993a) .  Major  Command  Dyna-METRIC 
microcomputer  Analysis  system  (MAJCOM-DMAS)  is  a  windows- 
based  microcomputer  version  of  Dyna-METRIC  version  4  but  it 
is  intended  for  more  sophisticated  scenarios.  It  provides 
access  to  a  much  larger  selection  of  the  version  4 
capabilities.  This  version  has  all  the  features  of  DMAS  but 
also  allows  multi-unit,  multi-echelon,  theater-level 
assessments  (DRC,  1993b) . 


Verification  and  Validation 


Model  verification  and  validation  are  important  aspects 
of  any  model  building  research.  Many  definitions  of  these 
terms  exist  in  the  literature.  Pritsker  (1986)  defines 
verification  as  "The  process  of  establishing  that  the 
computer  program  executes  as  intended.”  Khoshnevis  (1994) 
defines  it  as  a  "computer  implementation  of  the  model  that 
is  error-free, ”  and  he  points  out  it  is  not  concerned  with 
establishing  whether  the  model  is  reasonable.  Pegden  (1990) 
agrees  that  it  is  checking  the  model  to  see  that  it  performs 
as  expected  and  intended.  Law  and  Kelton  (1991)  see 
verification  as  a  debugging  process. 

Validation  is  defined  by  Pegden  (1990)  as  the  process 
of  raising  the  user' s  confidence  to  an  acceptable  level  such 
that  he  is  willing  to  use  inferences  drawn  from  the  model 
about  the  real  system.  Shannon  (1975)  agrees  with  this 
definition.  Pritsker  (1986)  calls  validation  the  process  of 
achieving  a  desired  accuracy  between  the  model  and  the  real 
system.  Khoshnevis  (1994)  defines  it  as  the  process  which 
establishes  that  the  model  and  the  input  data  represent  the 
important  aspects  of  the  modeled  system.  Gross  and  Harris 
(1985)  and  Lee,  Moore,  and  Taylor  (1990)  view  validation  as 
a  combination  of  the  verification  and  validation  processes 
while  Shannon  (1975)  credits  Fishman  and  Kiviat(1967)  with 
breaking  the  model  evaluation  process  into  three  steps: 


1)  Verification  -  that  the  model  runs  as  the  modeler 
intended, 

2)  Validation  -  that  there  is  agreement  in  the  behavior 
of  the  model  and  the  real  system, 

3)  Problem  Analysis  -  that  the  model  user  is  capable  of 
drawing  significant  inferences  from  the  model  results. 

The  question  of  verification  and  validation  boils  down 
to  answering  the  question,  does  the  model  represent 
reality?.  Specht  (1968)  warns  that  the  model  is  just  an 
"analog  of  reality"  and  may  not  represent  every  aspect  of 
reality.  He  stresses  that  the  important  thing  is  that  the 
model  outputs  answer  our  questions  in  an  appropriate  and 
valid  manner.  He  states  that  because  we  do  not  have 
complete  knowledge  the  best  we  can  hope  to  do  is  answer  the 
following  questions: 

1)  Can  the  model  describe  clearly  and  correctly  the 
known  facts  and  circumstances? 

2)  When  the  input  parameters  are  varies,  are  the 
results  consistent  and  reasonable? 

3)  How  does  the  model  handle  special  cases  where  we 
have  some  indication  of  the  outcome? 

4)  Can  it  assign  causes  to  known  effects  (Specht, 1968) . 
Shannon  (1975)  states  that  it  is  impossible  to  prove  that 
any  model  is  the  true  or  correct  model  of  a  system  but  adds 
that  this  is  seldom  important  since  we  are  primarily 
concerned  with  validating  the  insights  gained  from  the 
model.  He  stresses  that  these  insights  provide  the  model 


utility  not  the  model  structure  accuracy.  Specht  (1968) 
agrees.  He  says  we  should  not  be  upset  that  a  model  does 
not  look  like  the  real  thing  or  that  it  does  not  represent 
all  of  reality.  He  adds  that  several  models  for  the  same 
reality  may  be  valid  depending  on  the  questions  asked  and 
the  decisions  affected.  Pegden  (1990)  clarifies  this 
discussion  by  saying  that  verification  is  where  the  modeler 
gains  confidence  in  the  model  and  validation  is  where  the 
modeler  transfers  that  confidence  to  others. 

As  stated  above,  validation  is  the  process  of 
convincing  the  decision  maker  or  model  user  that  the  model 
accurately  reflects  the  real  system  for  the  user's  decision 
making  or  analysis  purposes.  Pritsker  (1986),  Gross  and 
Harris (1985)  and  Lee,  Moore,  and  Taylor  (1990)  all  agree 
that  validation  is  the  more  difficult  of  the  two  evaluation 
tasks.  The  ultimate  goal  of  any  model  is  to  aid  the 
decision  maker,  therefore,  the  modeler  would  like  to  test 
the  correctness  and  relevance  of  the  results.  Since  this  is 
not  always  possible,  says  Specht  (1968),  the  best  we  can 
sometimes  hope  for  is  to  be  honest.  Further,  claims  Quade 
(1968),  no  matter  how  hard  the  analyst  strives  to  maintain  a 
scientific  inquiry  or  to  follow  scientific  methods,  military 
systems  analysis  is  not  an  exact  science.  Although  it  may 
appear  rational,  analytical,  and  objective,  do  not  be 
fooled.  Human  judgment  is  used  in  the  analysis  for: 

1)  creating  the  analysis  design, 

2)  determining  the  relevant  factors. 


3)  determining  the  interactions  to  model  and  what  those 
to  leave  out, 

4)  choosing  the  alternatives  to  consider, 

5)  selecting  input  data, 

6)  analyzing  and  interpreting  results. 

Thus,  Quade  cautions,  judgment  and  intuition  are  infallible 
and,  as  such,  caution  is  advised  for  both  the  modeler  and 
the  decision  maker  to  avoid  biasing  the  model  results 
(Quade,  1968) .  Specht  gives  a  similar  warning: 

This  fact  -  that  judgment  and  intuition  and  guesswork 
are  embedded  in  a  model  -  should  be  remembered  when  we 
examine  the  results  that  come,  with  high  precision,  from  a 
model. (Specht,  1968) 

Verification  is  usually  done  by  using  a  manual  check  of 
the  calculations  claims  Pritsker  (1986) .  Gross  and  Harris 
(1985)  recommend  a  similar  method.  Pegden  (1990)  says  the 
best  support  for  verification  comes  from  proper  program 
design,  plus  clarity,  style,  and  ease  of  understanding.  He 
also  recommends  a  four  step  approach  to  verification  that 
includes  establishing  a  skeptical  frame  of  mind,  using 
outside  skeptics,  conducting  model  and  experimental  walk¬ 
throughs,  and  performing  test  runs.  As  part  of  this 
process,  Law  and  Kelton  (1991),  Khoshnevis  (1994)  and  Pegden 
(1990)  all  recommend  computer  animation  as  a  valuable 
verification  tool,  if  it  is  available.  Khoshnevis  (1994) 
states  that  models  usually  fail  to  operate  correctly  as  a 
result  of  coding  errors  or  logic  errors.  Coding  errors,  he 


says,  are  usually  the  easier  of  the  two  types  to  find 
because  they  stop  the  program' s  execution  process  and  are 
usually  located  by  the  compiler's  error  checking  system. 
Logic  errors,  on  the  other  hand,  are  much  more  difficult  to 
find  and  correct.  Both  Khoshnevis  (1994)  and  Pegden  (1990) 
provide  lists  of  the  most  common  error  types.  These  lists 
include  such  things  as  data  errors,  entity  flow  problems  or 
deadlocks,  problems  with  units  of  measurement,  and 
overwriting  model  variables.  They  both  also  suggest 
techniques  to  avoid  these  common  errors.  Law  and  Kelton 
(1991)  provide  8  different  techniques  for  verification  that 
include  modular  code  construction  and  testing,  debugging 
using  a  program  trace  function,  animation,  and  structured 
walk-throughs . 

Specht  (1968)  and  Law  and  Kelton  (1991)  each  present 
equivalent  three  step  approaches  to  model  validation.  They 
recommend: 

1)  testing  the  face  validity  of  the  model  -  These  tests 
determine  whether  a  model  seems  reasonable  to  experts 
familiar  with  the  system  under  study. 

2)  testing  the  model  assumptions  -  This  involves 
testing  quantitatively  the  assumptions  that  were  made  in  the 
model  early  in  development. 

3)  testing  the  reasonableness  of  the  input-output 
transformation  -  This  involves  testing  whether  or  not  the 
model  produces  results  similar  to  the  real  or  proposed 
system. 


% 


Steps  two  and  three  can  use  a  number  of  statistical 
techniques  and  methods  for  completing  these  steps.  These 
techniques  include  statistical  tests  of  means  and  variances, 
regression,  analysis  of  variance,  autocorrelation,  and  non- 
parametric  tests.  Shannon  (1975)  warns,  however,  that  each 
of  these  test  procedures  comes  with  its  own  set  of 
assumptions  that  must  be  considered.  Hillier  and  Lieberman 
(1986),  Pritsker  (1986),  Gross  and  Harris  (1985)  and  Pegden 
(1990)  all  agree  that  using  standard  statistical  tests  is 
the  way  to  go  if  data  exists  for  comparison.  If  data  does 
not  exist,  Hillier  and  Lieberman  (1986)  recommend  using  face 
value  testing  but  also  suggested  field  testing  to  collect 
data,  if  possible,  and  using  experts  in  the  field  to  perform 
sensitivity  analysis  on  the  model  under  a  variety  of 
different  scenarios.  They  point  out,  however,  that  field 
testing  is  frequently  costly  and  time-consuming  and 
therefore,  often  impractical.  Regardless,  they  emphasize 
the  importance  of  convincing  the  decision-maker  of  the 
credibility  of  the  model  for  decision-making  purposes.  Care 
must  be  taken  when  using  past  performance.  Pritsker  (1986) 
warns  that  past  performance  may  only  represent  one  sample 
and  not  necessarily  the  exact  answer.  Dalkey  (1968)  agrees 
and  adds  that  even  if  the  two  disagree,  the  model  may  still 
be  valid.  He  points  out  that  chance,  model  detail  and  a 
commander's  decisions  all  affect  historical  outcomes.  He 
emphasizes  that  a  major  factor  in  war  is  a  commander's 


decisions  and  presently  there  is  no  adequate  way  to  model 
these  decisions. 

Lee,  Moore  and  Taylor  (1990)  warn  that  simulation 
modeling  is  particularly  susceptible  to  the  garbage  in¬ 
garbage  out  syndrome.  Pritsker  (1986)  sees  the  validation 
process  as  a  way  of  answering  two  questions.  They  are: 

1)  What  is  the  inherent  variability  within  the  model? 

2)  What  can  be  inferred  about  the  real  system 
performance  from  the  model  performance? 

He  claims  the  first  question  deals  with  understanding  the 
model  and  assuring  the  model  operates  as  intended.  The 
second  deals  with  the  model's  usefulness.  The  first 
involves  obtaining  a  detailed  statistical  analysis  on  the 
precision  and  sensitivity  of  the  model.  The  second  question 
is  related  to  the  modeled  system  and,  therefore,  model 
dependent,  and  as  such  there  are  no  general  analysis  methods 
that  can  be  recommended  beyond  the  standard  statistical 
tools. 

Khoshnevis  (1994)  states  that  there  are  two  different 
approaches  to  model  validity.  Empiricism  and  Rationalism. 
Pegden  (1990)  adds  a  third  he  calls  Positive  Economics. 
Rationalism  is  an  approach  that  assumes  most  of  the 
underlying  assumptions  a  model  is  based  on  are  obviously 
true  and  therefore  in  no  need  of  proof.  Logical  deductions 
are  then  used  to  develop  the  model  and,  as  such,  if  the 
assumptions  and  logic  are  valid,  then  the  model  is  valid. 

The  empirical  approach  demands  that  every  assumption  and 


result  be  empirically  tested  and  validated.  No  assumption 
is  allowed  that  cannot  be  independently  tested  or  verified. 
The  Positive  Economics  approach  only  requires  that  the  model 
be  capable  of  predicting  the  future.  It  is  not  concerned 
with  validating  the  underlying  model  assumptions  or 
structure.  Thus,  if  the  model  has  good  predictive 
capability,  it  is  assumed  to  be  valid.  Although  Law  and 
Kelton  (1991)  and  Khoshnevis  (1994)  both  provided  several 
approaches  to  increasing  model  validity,  Pegden  (1990) 
provided  the  most  complete  list.  He  broke  these  approaches 
into  three  areas.  They  include  tests  for  reasonableness, 
tests  of  model  structure  and  data,  and  tests  of  model 
behavior.  Within  tests  for  reasonableness,  he  recommends 
checking  the  following: 

Continuity:  Small  changes  in  the  input  data  should 
result  in  small  but  appropriate  changes  in  the  decision 
variables. 

Consistency:  Similar  runs  of  the  model  should  result 
in  similar  outcomes. 

Degeneracy:  If  model  features  are  removed,  the 
decision  variables  should  reflect  their  removal. 

Absurd  Conditions:  This  has  two  parts.  (1)  If  the 
modeler  provides  unusual  data  input,  the  model  should  not 
give  unusual  or  unpredictable  results.  (2)  The  model  should 
never  generate  absurd  or  impossible  situations. 


For  testing  the  model  structure  and  data,  Pegden  (1990) 
proposes  these  tests: 

Face  validity:  This  is  accomplished  by  asking  experts 
of  the  modeled  system  whether  the  model's  behavior  seems 
reasonable. 

Parameters  and  Relationships:  This  area  includes  tests 
of  assumptions  concerning  parameter  values  and  variable 
relationships  and  typically  involves  statistical  tests  such 
as  means,  variances,  regression  analysis,  and  goodness  of 
fit  tests. 

Structural  and  Boundary  Verification:  The  key  here  is 
to  ensure  that  the  structure  of  the  model  does  not  clearly 
contradict  the  real  system. 

Sensitivity  Analysis:  This  is  done  by  varying  the 
model  input  parameters  and  checking  the  impact  of  these 
changes  to  the  model's  outputs.  This  should  give  us  some 
idea  how  sensitive  the  model  is  to  small  changes  in  the 
input  parameters. 

Also,  he  suggests  a  number  of  tests  for  investigating  model 
behavior : 

Behavior  Comparison:  This  involves  comparing  the  model 
output  to  the  real  system  results .  He  lists  a  number  of 
statistical  tests  that  are  available  including  the  Chi- 
Square  test,  Kolmgorov-Smirnov  test,  and  regression 
analysis. 


Symptom  Generation:  These  tests  take  different  forms 
but  answer  questions  like: 

-  Can  the  model  produce  the  same  difficulties  that 
show  up  in  the  real  world? 

Does  it  produce  the  same  results  after  a  change  as 
the  real  system  did  after  similar  changes? 

Behavior  Anomaly:  If  the  model  gives  results  that 
conflict  with  the  modeler's  expectations,  can  the  modeler 
find  examples  of  this  behavior  in  the  actual  system.  If  he 
cannot,  there  may  be  a  problem  with  the  model. 

Behavior  Prediction:  Use  the  model  to  predict  system 
performance  during  field  tests  by  controlling  inputs  to  both 
the  model  and  the  actual  system. 


Finally,  Law  and  Kelton  (1991),  Shannon  (1975),  and 
Pegden  (1990)  stress  that  verification  and  validation  are 
ongoing  tasks  and  should  not  be  left  until  the  end  of  the 
project,  to  be  done  if  time  and  money  permit.  Figure  1 
illustrates  the  continuous  roles  verification  and  validation 
play  in  the  model  development  process.  This  section 
concludes  with  a  thought  by  Shannon  (1975) : 


The  question  of  validation  is  thus  two-faced: 
determining  that  the  model  behaves  in  the 
same  fashion  as  the  real  system;  validating 
that  inferences  drawn  from  the  experiments 
with  the  model  are  valid  or  correct.  In 
concept,  both  these  points  resolve  themselves 
to  the  standard  decision  problem  of  balancing 
the  cost  of  each  action  against  the  value  of 


the 


the  increased  information  and 
consequences  of  erroneous  conclusions. 
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Chapter  III:  METHODOLOGY 


The  research  can  be  broken  into  three  general  parts. 
These  areas  include  data  gathering,  model  development  and 
validation,  and  a  model  output  analysis  plan.  The  research 
will  identify  current  shortcomings  of  the  current  Air  Force 
combat  aircraft  assessment  techniques  which  center  around 
the  use  of  the  Dyna-METRIC  model.  Then  it  will  consider 
several  analytic  and  simulation-based  techniques  to 
determine  a  better  method.  My  approach  in  each  area  will  be 
discussed  below. 

Data  gathering  or  more  accurately  information  gathering 
will  occur  in  several  ways.  Aircraft  demand  and  repair 
information  will  generally  come  from  HQ  ACC/LGSW  although  HQ 
AFMC  is  a  secondary  source.  Part  of  this  research  will 
consider  what  data  elements  are  actually  necessary  for  these 
assessments.  Current  information  on  the  model  capabilities 
comes  from  a  wide  variety  of  sources.  In  addition  to  the 
two  sources  listed  above,  I  expect  to  receive  information  on 
the  model  from  HQ  AF/LEYS,  Logistics  Management  Institute 
(LMI) ,  Dynamics  Research  Corporation  (DRC)  and  RAND 
Corporation.  Each  of  these  groups  has  an  active  interest 
and  involvement  in  the  current  model  and  future  assessment 
developments.  Military  regulations  and  manuals  will  also 
provide  guidance.  Finally,  current  literature  provides  an 
additional  source  of  information  on  the  current  assessment 


method  and  knowledge  about  other  techniques  which  could  be 
adapted  or  developed  into  a  method  for  addressing  this 
problem.  Various  models  for  comparison  with  the  research 
model  will  come  from  RAND,  DRC,  or  HQ  ACC/LGSW.  The  data 
gathering  efforts  should  feed  the  methodology  development 
and  data  analysis  section  of  this  work. 

Methodology  development  and  validation  represent  the 
heart  of  this  research  project.  This  involves  a  number  of 
steps.  A  clear  understanding,  explanation,  and 
demonstration  of  the  problems  and  shortfalls  of  the  current 
assessment  methodology  must  be  established.  This  can  be 
done  through  a  thorough  literature  review  and  discussions 
with  the  practitioners  in  the  field.  Based  on  this  process 
understanding,  other  possible  methods  and  modeling 
techniques  can  be  investigated  as  improvements  to  the 
current  process. 

Several  approaches  appear  promising  as  starting  points 
for  this  research.  The  first  is  based  on  Jacksonian 
networks  (Jackson,  1957  and  Jackson,  1963) .  Gross  et  al. 
(1983)  used  this  networking  method  to  develop  a  technique  to 
optimize  sparing  levels  and  repair  channels.  It  is  an 
extension  of  earlier  work  done  by  Mirasol  (1964) .  Mirasol 
built  a  model  which  considered  a  single-echelon,  single¬ 
repair  shop,  infinite  calling  population,  finite  repair 
capacity  system.  Gross  added  to  this  work  by  considering  a 
finite  population  of  operating  times  and  a  two  echelon 
repair  system.  The  drawbacks  with  this  method  are  that  it 


only  considers  steady  state  conditions  and  it  was  built  to 
optimize  sparing  levels  based  on  cost  not  to  determine 
availability  based  on  sparing  levels.  Another  promising 
approach  uses  a  method  developed  by  Sherbrooke  (1992)  for 
the  NASA  space  station  Freedom.  It  is  conceptually  similar 
in  many  ways  to  the  METRIC-type  models  but  does  not  use 
Palm's  theorem  because  it  does  not  assume  that  the  shapes  of 
the  repair  distributions  are  unimportant.  It  also  does  not 
use  an  infinite  population  assumption  to  distribute 
backorders.  One  drawback  of  this  method  is  that  it  still 
assumes  an  infinite  calling  population  for  demands. 

Kaplan (1989) ,  however,  developed  a  model  similar  to  Isaacson 
(1988)  but  includes  the  finite  calling  population  assumption 
in  his  model.  This  technique  could  be  adapted  into 
Sherbrooke's  model  (1992)  or  possibly  extended  as  a  primary 
method. 

When  a  technique  has  been  selected,  it  will  be 
developed  and  validated.  Model  validation,  defined  as 
raising  the  decision-maker' s  confidence  to  an  acceptable 
level,  will  include  a  number  of  steps  (Shannon,  1975) . 
Checking  the  face  or  structure  and  assumptions  validity  of 
the  method  and  checking  an  input-output  transformation 
analysis  with  the  current  method  and  with  a  real  deployment 
are  two  of  the  steps.  The  structure  and  assumptions  testing 
will  be  an  iterative  process  involving  the  field  experts. 

As  developments  are  made,  they  will  be  checked  with  the 
agencies  mentioned  earlier  for  credibility  and  accuracy. 


When  these  groups  are  satisfied  that  the  method  is 
structurally  sound,  this  portion  of  the  validation  will  be 
complete.  Once  the  method  has  been  developed  and  the 
structure  validated,  it  can  be  compared  with  the  current 
method  and  reality.  The  data  analysis  section  deals  with 
this  part  of  the  research.  The  users  will  provide  the 
expert  input  for  face-value  and  output  validity  tests. 

The  data  analysis  plan  depends  on  the  new  assessment 
method  selected  and  results  from  the  current  method  and  a 
live  aircraft  test.  The  current  method  uses  number  of 
sorties  flown  during  the  deployment  period  and  the  number  of 
combat  ready  aircraft  available  at  the  end  of  the  deployment 
period  as  the  measures  of  merit.  Since  these  are  the  Air 
Force  standard  measures,  this  study  will  also  use  them  as 
the  basis  of  method  comparison.  HQ  ACC  plans  an  actual 
deployment  test  of  B-l  aircraft  to  Roswell,  NM  in  November 
1994.  The  data  collected  from  this  test  will  form  the  basis 
of  the  live  data  set.  To  ensure  a  reliable  comparison 
between  this  actual  data,  the  current  method,  and  the  new 
method,  the  two  modeling  methods  will  use  the  same  scenario 
and  assumptions  that  are  being  used  during  the  B-l  test. 
Using  this  input  data,  the  two  models  will  be  run  and  their 
output  collected.  Based  on  the  output  from  all  three 
sources,  a  series  of  two-sided  Hypothesis  tests  will  be 
created  to  test  the  differences  between  the  data  sets.  The 
null  hypothesis  will  be  that  there  is  no  difference  between 
the  data  sets  while  the  alternative  hypothesis  will  be  that 


there  is  a  statistically  significant  difference  in  the  data. 
These  hypotheses  will  be  tested  at  a  95%  significance  level. 
As  a  result  of  these  tests,  a  discussion  of  similarities  and 
differences  between  the  data  will  be  made.  Also, 
conclusions  and  recommendations  for  future  work  and  possible 
implementation  of  this  method  in  the  Air  Force  will  be  made. 
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